52 research outputs found
Optimistic Value Iteration
Markov decision processes are widely used for planning and verification in settings that combine controllable or adversarial choices with probabilistic behaviour. The standard analysis algorithm, value iteration, only provides lower bounds on infinite-horizon probabilities and rewards. Two âsoundâ variations, which also deliver an upper bound, have recently appeared. In this paper, we present a new sound approach that leverages value iterationâs ability to usually deliver good lower bounds: we obtain a lower bound via standard value iteration, use the result to âguessâ an upper bound, and prove the latterâs correctness. We present this optimistic value iteration approach for computing reachability probabilities as well as expected rewards. It is easy to implement and performs well, as we show via an extensive experimental evaluation using our implementation within the mcsta model checker of the Modest Toolset
A Hierarchy of Scheduler Classes for Stochastic Automata
Stochastic automata are a formal compositional model for concurrent
stochastic timed systems, with general distributions and non-deterministic
choices. Measures of interest are defined over schedulers that resolve the
nondeterminism. In this paper we investigate the power of various theoretically
and practically motivated classes of schedulers, considering the classic
complete-information view and a restriction to non-prophetic schedulers. We
prove a hierarchy of scheduler classes w.r.t. unbounded probabilistic
reachability. We find that, unlike Markovian formalisms, stochastic automata
distinguish most classes even in this basic setting. Verification and strategy
synthesis methods thus face a tradeoff between powerful and efficient classes.
Using lightweight scheduler sampling, we explore this tradeoff and demonstrate
the concept of a useful approximative verification technique for stochastic
automata
Confluence reduction for Markov automata
Markov automata are a novel formalism for specifying systems exhibiting nondeterminism, probabilistic choices and Markovian rates. Recently, the process algebra MAPA was introduced to efficiently model such systems. As always, the state space explosion threatens the analysability of the models generated by such specifications. We therefore introduce confluence reduction for Markov automata, a powerful reduction technique to keep these models small. We define the notion of confluence directly on Markov automata, and discuss how to syntactically detect confluence on the MAPA language as well. That way, Markov automata generated by MAPA specifications can be reduced on-the-fly while preserving divergence-sensitive branching bisimulation. Three case studies demonstrate the significance of our approach, with reductions in analysis time up to an order of magnitude
Explicit Model Checking of Very Large MDP using Partitioning and Secondary Storage
The applicability of model checking is hindered by the state space explosion
problem in combination with limited amounts of main memory. To extend its
reach, the large available capacities of secondary storage such as hard disks
can be exploited. Due to the specific performance characteristics of secondary
storage technologies, specialised algorithms are required. In this paper, we
present a technique to use secondary storage for probabilistic model checking
of Markov decision processes. It combines state space exploration based on
partitioning with a block-iterative variant of value iteration over the same
partitions for the analysis of probabilistic reachability and expected-reward
properties. A sparse matrix-like representation is used to store partitions on
secondary storage in a compact format. All file accesses are sequential, and
compression can be used without affecting runtime. The technique has been
implemented within the Modest Toolset. We evaluate its performance on several
benchmark models of up to 3.5 billion states. In the analysis of time-bounded
properties on real-time models, our method neutralises the state space
explosion induced by the time bound in its entirety.Comment: The final publication is available at Springer via
http://dx.doi.org/10.1007/978-3-319-24953-7_1
PrIC3: Property Directed Reachability for MDPs
IC3 has been a leap forward in symbolic model checking. This paper proposes
PrIC3 (pronounced pricy-three), a conservative extension of IC3 to symbolic
model checking of MDPs. Our main focus is to develop the theory underlying
PrIC3. Alongside, we present a first implementation of PrIC3 including the key
ingredients from IC3 such as generalization, repushing, and propagation
A Comparison of Time- and Reward-Bounded Probabilistic Model Checking Techniques
In the design of probabilistic timed systems, requirements concerning behaviour that occurs within a given time or energy budget are of central importance. We observe that model-checking such requirements for probabilistic timed automata can be reduced to checking reward-bounded properties on Markov decision processes. This is traditionally implemented by unfolding the model according to the bound, or by solving a sequence of linear programs. Neither scales well to large models. Using value iteration in place of linear programming achieves scalability but accumulates approximation error. In this paper, we correct the value iteration-based scheme, present two new approaches based on scheduler enumeration and state elimination, and compare the practical performance and scalability of all techniques on a number of case studies from the literature. We show that state elimination can significantly reduce runtime for large models or high bounds
- âŠ